Rater Drift in Constructed Response Scoring via Latent Class Signal Detection Theory and Item Response Theory

The use of constructed response (CR) items or performance tasks to assess test takers' ability has grown tremendously over the past decade. Examples of CR items in psychological and educational measurement range from essays, works of art, and admissions interviews. However, unlike multiple-choice (MC) items that have predetermined options, CR items require test takers to construct their own answer. As such, they require the judgment of multiple raters that are subject to differences in perception and prior knowledge of the material being evaluated. As with any scoring procedure, the scores assigned by raters must be comparable over time and over different test administrations and forms; in other words, scores must be reliable and valid for all test takers, regardless of when an individual takes the test. This study examines how longitudinal patterns or changes in rater behavior affect model-based classification accuracy. Rater drift refers to changes in rater behavior across different test administrations. Prior research has found evidence of drift. Rater behavior in CR scoring is examined using two measurement models - latent class signal detection theory (SDT) and item response theory (IRT) models. Rater effects (e.g., leniency and strictness) are partly examined with simulations, where the ability of different models to capture changes in rater behavior is studied. Drift is also examined in two real-world large scale tests: teacher certification test and high school writing test. These tests use the same set of raters for long periods of time, where each rater's scoring is examined on a monthly basis. Results from the empirical analysis showed that rater models were effective to detect changes in rater behavior over testing administrations in real-world data. However, there were differences in rater discrimination between the latent class SDT and IRT models. Simulations were used to examine the effect of rater drift on classification accuracy and on differences between the latent class SDT and IRT models. Changes in rater severity had only a minimal effect on classification. Rater discrimination had a greater effect on classification accuracy. This study also found that IRT models detected changes in rater severity and in rater discrimination even when data were generated from the latent class SDT model. However, when data were non-normal, IRT models underestimated rater discrimination, which may lead to incorrect inferences on the precision of raters. These findings provide new and important insights into CR scoring and issues that emerge in practice, including methods to improve rater training

Assessing the Reliability and Validity of the Evacuation Support Decision Tool

This study examines the reliability and validity of the Evacuation Decision Support Tool (EDST). The EDST is designed to provide healthcare facilities, emergency managers, and other agencies with a systematic process with which to evaluate and guide “evacuation” versus “shelter in place” decision making for a variety of “all hazards” situations. The EDST is comprised of 7 items that assess “threat” and 9 items that measure “consequences” of a situation. The tool was designed to provide users with a decision on whether to remain, prepare, or evacuate from a healthcare facility. To date, there has not been a study that examined psychometric properties of any evacuation decision tool, including the EDST

The Positivism Paradigm of Research.

Research paradigms guide scientific discoveries through their assumptions and principles. Understanding paradigm-specific assumptions helps illuminate the quality of findings that support scientific studies and identify gaps in generating sound evidence. This article focuses on the research paradigm of positivism, examining its definition, history, and assumptions (ontology, epistemology, axiology, methodology, and rigor). Positivism is aligned with the hypothetico-deductive model of science that builds on verifying a priori hypotheses and experimentation by operationalizing variables and measures; results from hypothesis testing are used to inform and advance science. Studies aligned with positivism generally focus on identifying explanatory associations or causal relationships through quantitative approaches, where empirically based findings from large sample sizes are favored-in this regard, generalizable inferences, replication of findings, and controlled experimentation have been principles guiding positivist science. Criteria for evaluating the quality of positivist research are discussed. An example from health professions education is provided to guide positivist thinking in study design and implementation

Do Shared Barriers When Reporting to Work During an Influenza Pandemic Influence Hospital Workers’ Willingness to Work? A Multilevel Framework

Objective Characteristics associated with interventions and barriers that influence health care workers’ willingness to report for duty during an influenza pandemic were identified. Additionally, this study examined whether workers who live in proximal geographic regions shared the same barriers and would respond to the same interventions. Methods Hospital employees (n=2965) recorded changes in willingness to work during an influenza pandemic on the basis of interventions aimed at mitigating barriers. Distance from work, hospital type, job role, and family composition were examined by clustering the effects of barriers from reporting for duty and region of residence. Results Across all workers, providing protection for the family was the greatest motivator for willingness to work during a pandemic. Respondents who expressed the same barriers and lived nearby shared common responses in their willingness to work. Younger employees and clinical support staff were more receptive to interventions. Increasing distance from home to work was significantly associated with a greater likelihood to report to work for employees who received time off. Conclusions Hospital administrators should consider the implications of barriers and areas of residence on the disaster response capacity of their workforce. Our findings underscore communication and development of preparedness plans to improve the resilience of hospital workers to mitigate absenteeism (Disaster Med Public Health Preparedness. 2015;9:175-185)

Mitigating absenteeism in hospital workers during a pandemic

Objectives: An influenza pandemic, as with any disaster involving contagion or contamination, has the potential to influence the number of health care employees who will report for duty. Our project assessed the uptake of proposed interventions to mitigate absenteeism in hospital workers during a pandemic. Methods: Focus groups were followed by an Internet-based survey of a convenience sample frame of 17,000 hospital workers across 5 large urban facilities. Employees were asked to select their top barrier to reporting for duty and to score their willingness to work before and after a series of interventions were offered to mitigate it. Results: Overall, 2864 responses were analyzed. Safety concerns were the most frequently cited top barrier to reporting for work, followed by issues of dependent care and transportation. Significant increases in employee willingness to work scores were observed from mitigation strategies that included preferential access to antiviral medication or personal protective equipment for the employee as well as their immediate family. Conclusions: The knowledge base on workforce absenteeism during disasters is growing, although in general this issue is underrepresented in emergency planning efforts. Our data suggest that a mitigation strategy that includes options for preferential access to either antiviral therapy, protective equipment, or both for the employee as well as his or her immediate family will have the greatest impact. These findings likely have import for other disasters involving contamination or contagion, and in critical infrastructure sectors beyond health care

Programmable spectral shaping to improve the measurement precision of frequency comb mode-resolved spectral interferometric ranging

Comb-mode resolved spectral domain interferometry (CORE-SDI), which is capable of measuring length of kilometers or more with precision on the order of nanometers, is considered to be a promising technology for next-generation length standards, replacing laser displacement interferometers. In this study, we aim to improve the measurement precision of CORE-SDI using programmable spectral shaping. We report the generation of effectively broad and symmetric light sources through the programmable spectral shaping. The light source used here was generated by the spectrally-broadened electro-optic comb with a repetition rate of 17.5 GHz. Through the programmable spectral shaping, the optical spectrum was flattened within 1 dB, resulting in a square-shaped optical spectrum. As a result, the 3-dB spectral width was extended from 1.15 THz to 6.7 THz. We performed a comparison between the measurement results of various spectrum shapes. We confirmed an improvement in the measurement precision from 69 nm to 6 nm, which was also corroborated by numerical simulations. We believe that this study on enhancing the measurement precision of CORE-SDI through the proposed spectral shaping will make a significant contribution to reducing the measurement uncertainty of future CORE-SDI systems, thereby advancing the development of next-generation length standards.Comment: 22 pages, 10 figure